Collaboration and Competition¶

You are welcome to use this coding environment to train your agent for the project. Follow the instructions below to get started!

1. Start the Environment¶

Run the next code cell to install a few packages. This line will take a few minutes to run!

!pip -q install ./python

# load basic packages
import os
import sys
sys.path.append('files')

# load, torch, datetime, queue and matlotlib
import torch
from datetime import datetime
from collections import deque
import matplotlib.pyplot as plt

def get_time(): return datetime.now().strftime("%Y-%d-%d_%H-%M-%S")

# update objects|classes when we write code
%load_ext autoreload
%autoreload 2

# show plots
%matplotlib inline

# install weights and biases
!{sys.executable} -m pip install wandb
!{sys.executable} -m pip install bayesian-optimization

tensorflow 1.7.1 has requirement numpy>=1.13.3, but you'll have numpy 1.12.1 which is incompatible.
ipython 6.5.0 has requirement prompt-toolkit<2.0.0,>=1.0.15, but you'll have prompt-toolkit 2.0.10 which is incompatible.
Collecting wandb
  Downloading https://files.pythonhosted.org/packages/79/b7/8f54fbb4b0490764657b89ef760332d0c64e5090d3c22880f86c8471817f/wandb-0.8.18-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 16.2MB/s ta 0:00:01
Collecting subprocess32>=3.5.3 (from wandb)
  Downloading https://files.pythonhosted.org/packages/32/c8/564be4d12629b912ea431f1a50eb8b3b9d00f1a0b1ceff17f266be190007/subprocess32-3.5.4.tar.gz (97kB)
    100% |████████████████████████████████| 102kB 22.7MB/s a 0:00:01
Collecting nvidia-ml-py3>=7.352.0 (from wandb)
  Downloading https://files.pythonhosted.org/packages/6d/64/cce82bddb80c0b0f5c703bbdafa94bfb69a1c5ad7a79cff00b482468f0d3/nvidia-ml-py3-7.352.0.tar.gz
Collecting watchdog>=0.8.3 (from wandb)
  Downloading https://files.pythonhosted.org/packages/bb/e3/5a55d48a29300160779f0a0d2776d17c1b762a2039b36de528b093b87d5b/watchdog-0.9.0.tar.gz (85kB)
    100% |████████████████████████████████| 92kB 27.4MB/s ta 0:00:01
Collecting gql>=0.1.0 (from wandb)
  Downloading https://files.pythonhosted.org/packages/aa/9c/2933b7791210e00f5c26a6243198cc03af9132c29cf85e4c22cb007f171e/gql-0.1.0.tar.gz
Collecting sentry-sdk>=0.4.0 (from wandb)
  Downloading https://files.pythonhosted.org/packages/b4/48/a38ff26af174ef2d03c5754c93b7a18818951734fdaa53bf3c2b80e497e7/sentry_sdk-0.13.5-py2.py3-none-any.whl (91kB)
    100% |████████████████████████████████| 92kB 24.2MB/s ta 0:00:01
Collecting Click>=7.0 (from wandb)
  Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)
    100% |████████████████████████████████| 81kB 21.3MB/s ta 0:00:01
Requirement already satisfied: requests>=2.0.0 in /opt/conda/lib/python3.6/site-packages (from wandb) (2.18.4)
Collecting graphql-core<3.0.0 (from wandb)
  Downloading https://files.pythonhosted.org/packages/6a/11/bc4a7eb440124271289d93e4d208bd07d94196038fabbe2a52435a07d3d3/graphql_core-2.2.1-py2.py3-none-any.whl (250kB)
    100% |████████████████████████████████| 256kB 21.6MB/s ta 0:00:01
Collecting GitPython>=1.0.0 (from wandb)
  Downloading https://files.pythonhosted.org/packages/20/8c/4543981439d23c4ff65b2e62dddd767ebc84a8e664a9b67e840d1e2730d3/GitPython-3.0.5-py3-none-any.whl (455kB)
    100% |████████████████████████████████| 460kB 20.6MB/s ta 0:00:01
Requirement already satisfied: psutil>=5.0.0 in /opt/conda/lib/python3.6/site-packages (from wandb) (5.4.0)
Collecting docker-pycreds>=0.4.0 (from wandb)
  Downloading https://files.pythonhosted.org/packages/f5/e8/f6bd1eee09314e7e6dee49cbe2c5e22314ccdb38db16c9fc72d2fa80d054/docker_pycreds-0.4.0-py2.py3-none-any.whl
Collecting configparser>=3.8.1 (from wandb)
  Downloading https://files.pythonhosted.org/packages/7a/2a/95ed0501cf5d8709490b1d3a3f9b5cf340da6c433f896bbe9ce08dbe6785/configparser-4.0.2-py2.py3-none-any.whl
Collecting shortuuid>=0.5.0 (from wandb)
  Downloading https://files.pythonhosted.org/packages/80/d7/2bfc9332e68d3e15ea97b9b1588b3899ad565120253d3fd71c8f7f13b4fe/shortuuid-0.5.0.tar.gz
Requirement already satisfied: python-dateutil>=2.6.1 in /opt/conda/lib/python3.6/site-packages (from wandb) (2.6.1)
Requirement already satisfied: six>=1.10.0 in /opt/conda/lib/python3.6/site-packages (from wandb) (1.11.0)
Requirement already satisfied: PyYAML>=3.10 in /opt/conda/lib/python3.6/site-packages (from watchdog>=0.8.3->wandb) (3.12)
Collecting argh>=0.24.1 (from watchdog>=0.8.3->wandb)
  Downloading https://files.pythonhosted.org/packages/06/1c/e667a7126f0b84aaa1c56844337bf0ac12445d1beb9c8a6199a7314944bf/argh-0.26.2-py2.py3-none-any.whl
Collecting pathtools>=0.1.1 (from watchdog>=0.8.3->wandb)
  Downloading https://files.pythonhosted.org/packages/e7/7f/470d6fcdf23f9f3518f6b0b76be9df16dcc8630ad409947f8be2eb0ed13a/pathtools-0.1.2.tar.gz
Collecting promise>=0.4.0 (from gql>=0.1.0->wandb)
  Downloading https://files.pythonhosted.org/packages/5a/81/221d09d90176fd90aed4b530e31b8fedf207385767c06d1d46c550c5e418/promise-2.2.1.tar.gz
Requirement already satisfied: urllib3>=1.10.0 in /opt/conda/lib/python3.6/site-packages (from sentry-sdk>=0.4.0->wandb) (1.22)
Requirement already satisfied: certifi in /opt/conda/lib/python3.6/site-packages (from sentry-sdk>=0.4.0->wandb) (2019.6.16)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from requests>=2.0.0->wandb) (3.0.4)
Requirement already satisfied: idna<2.7,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests>=2.0.0->wandb) (2.6)
Collecting rx<3,>=1.6 (from graphql-core<3.0.0->wandb)
  Downloading https://files.pythonhosted.org/packages/33/0f/5ef4ac78e2a538cc1b054eb86285fe0bf7a5dbaeaac2c584757c300515e2/Rx-1.6.1-py2.py3-none-any.whl (179kB)
    100% |████████████████████████████████| 184kB 24.3MB/s ta 0:00:01
Collecting gitdb2>=2.0.0 (from GitPython>=1.0.0->wandb)
  Downloading https://files.pythonhosted.org/packages/03/6c/99296f89bad2ef85626e1df9f677acbee8885bb043ad82ad3ed4746d2325/gitdb2-2.0.6-py2.py3-none-any.whl (63kB)
    100% |████████████████████████████████| 71kB 18.6MB/s ta 0:00:01
Collecting smmap2>=2.0.0 (from gitdb2>=2.0.0->GitPython>=1.0.0->wandb)
  Downloading https://files.pythonhosted.org/packages/55/d2/866d45e3a121ee15a1dc013824d58072fd5c7799c9c34d01378eb262ca8f/smmap2-2.0.5-py2.py3-none-any.whl
Building wheels for collected packages: subprocess32, nvidia-ml-py3, watchdog, gql, shortuuid, pathtools, promise
  Running setup.py bdist_wheel for subprocess32 ... done
  Stored in directory: /root/.cache/pip/wheels/68/39/1a/5e402bdfdf004af1786c8b853fd92f8c4a04f22aad179654d1
  Running setup.py bdist_wheel for nvidia-ml-py3 ... done
  Stored in directory: /root/.cache/pip/wheels/e4/1d/06/640c93f5270d67d0247f30be91f232700d19023f9e66d735c7
  Running setup.py bdist_wheel for watchdog ... done
  Stored in directory: /root/.cache/pip/wheels/61/1d/d0/04cfe495619be2095eb8d89a31c42adb4e42b76495bc8f784c
  Running setup.py bdist_wheel for gql ... done
  Stored in directory: /root/.cache/pip/wheels/a7/8d/65/a3247f500d675d80a01e4d2f0ee44fe99f1faef575bc2a1664
  Running setup.py bdist_wheel for shortuuid ... done
  Stored in directory: /root/.cache/pip/wheels/3f/eb/fd/69e5177f67b505e44acbd1aedfbe44b91768ee0c4cd5636576
  Running setup.py bdist_wheel for pathtools ... done
  Stored in directory: /root/.cache/pip/wheels/0b/04/79/c3b0c3a0266a3cb4376da31e5bfe8bba0c489246968a68e843
  Running setup.py bdist_wheel for promise ... done
  Stored in directory: /root/.cache/pip/wheels/92/84/9f/75e2235effae0e1c5a5c0626a503e532bbffcb7e79e672b606
Successfully built subprocess32 nvidia-ml-py3 watchdog gql shortuuid pathtools promise
Installing collected packages: subprocess32, nvidia-ml-py3, argh, pathtools, watchdog, promise, rx, graphql-core, gql, sentry-sdk, Click, smmap2, gitdb2, GitPython, docker-pycreds, configparser, shortuuid, wandb
  Found existing installation: click 6.7
    Uninstalling click-6.7:
      Successfully uninstalled click-6.7
Successfully installed Click-7.0 GitPython-3.0.5 argh-0.26.2 configparser-4.0.2 docker-pycreds-0.4.0 gitdb2-2.0.6 gql-0.1.0 graphql-core-2.2.1 nvidia-ml-py3-7.352.0 pathtools-0.1.2 promise-2.2.1 rx-1.6.1 sentry-sdk-0.13.5 shortuuid-0.5.0 smmap2-2.0.5 subprocess32-3.5.4 wandb-0.8.18 watchdog-0.9.0
Collecting bayesian-optimization
  Downloading https://files.pythonhosted.org/packages/72/0c/173ac467d0a53e33e41b521e4ceba74a8ac7c7873d7b857a8fbdca88302d/bayesian-optimization-1.0.1.tar.gz
Requirement already satisfied: numpy>=1.9.0 in /opt/conda/lib/python3.6/site-packages (from bayesian-optimization) (1.12.1)
Requirement already satisfied: scipy>=0.14.0 in /opt/conda/lib/python3.6/site-packages (from bayesian-optimization) (0.19.1)
Requirement already satisfied: scikit-learn>=0.18.0 in /opt/conda/lib/python3.6/site-packages (from bayesian-optimization) (0.19.1)
Building wheels for collected packages: bayesian-optimization
  Running setup.py bdist_wheel for bayesian-optimization ... done
  Stored in directory: /root/.cache/pip/wheels/1d/0d/3b/6b9d4477a34b3905f246ff4e7acf6aafd4cc9b77d473629b77
Successfully built bayesian-optimization
Installing collected packages: bayesian-optimization
Successfully installed bayesian-optimization-1.0.1

A tool which keep track of the performance and hyperparameters

os.popen('wandb login b4325006f2643dxxxxxxx').read()
import wandb

The environment is already saved in the Workspace and can be accessed at the file path provided below.

from unityagents import UnityEnvironment
import numpy as np

env = UnityEnvironment(file_name="/data/Tennis_Linux_NoVis/Tennis")

INFO:unityagents:
'Academy' started successfully!
Unity Academy name: Academy
        Number of Brains: 1
        Number of External Brains : 1
        Lesson number : 0
        Reset Parameters :
		
Unity brain name: TennisBrain
        Number of Visual Observations (per agent): 0
        Vector Observation space type: continuous
        Vector Observation space size (per agent): 8
        Number of stacked Vector Observation: 3
        Vector Action space type: continuous
        Vector Action space size (per agent): 2
        Vector Action descriptions: ,

Environments contain brains which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python.

# get the default brain
brain_name = env.brain_names[0]
brain = env.brains[brain_name]

2. Examine the State and Action Spaces¶

Run the code cell below to print some information about the environment.

# reset the environment
env_info = env.reset(train_mode=True)[brain_name]

# number of agents 
num_agents = len(env_info.agents)
print('Number of agents:', num_agents)

# size of each action
action_size = brain.vector_action_space_size
print('Size of each action:', action_size)

# examine the state space 
states = env_info.vector_observations
state_size = states.shape[1]
print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))
print('The state for the first agent looks like:', states[0])

Number of agents: 2
Size of each action: 2
There are 2 agents. Each observes a state with length: 24
The state for the first agent looks like: [ 0.          0.          0.          0.          0.          0.          0.
  0.          0.          0.          0.          0.          0.          0.
  0.          0.         -6.65278625 -1.5        -0.          0.
  6.83172083  6.         -0.          0.        ]

3. Take Random Actions in the Environment¶

In the next code cell, you will learn how to use the Python API to control the agent and receive feedback from the environment.

Note that in this coding environment, you will not be able to watch the agents while they are training, and you should set train_mode=True to restart the environment.

for i in range(5):                                         # play game for 5 episodes
    env_info = env.reset(train_mode=False)[brain_name]     # reset the environment    
    states = env_info.vector_observations                  # get the current state (for each agent)
    scores = np.zeros(num_agents)                          # initialize the score (for each agent)
    while True:
        actions = np.random.randn(num_agents, action_size) # select an action (for each agent)
        actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1
        env_info = env.step(actions)[brain_name]           # send all actions to tne environment
        next_states = env_info.vector_observations         # get next state (for each agent)
        rewards = env_info.rewards                         # get reward (for each agent)
        dones = env_info.local_done                        # see if episode finished
        scores += env_info.rewards                         # update the score (for each agent)
        states = next_states                               # roll over states to next time step
        if np.any(dones):                                  # exit loop if episode finished
            break
    print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))

Total score (averaged over agents) this episode: -0.004999999888241291
Total score (averaged over agents) this episode: -0.004999999888241291
Total score (averaged over agents) this episode: 0.04500000085681677
Total score (averaged over agents) this episode: 0.04500000085681677
Total score (averaged over agents) this episode: -0.004999999888241291

.
.
.
.
.
.
.
.
.
.

PROJECT 3: Collaboration and Competition!¶

This project/report explains briefly, (together with the comments in the code) how to solve the Collaboration and Competition problem of Udacity's Deep Reinforcement Learning Nanodegree Program.
In this project we've to train two agents which are playing tennis with each other. The goal of this project is, to keep the tennis ball in the air as long a possible. Each time when an agent is able to move the ball over the net, we receive a reward of +0.1. However when the tennis bal drops on the floor, we receive a reward of -0.01. Thus, the goal of each agent is to keep the ball in play.

How to start?¶

Basically we start were we left off and that is the Continuous Control project. In that project we implemented the DDPG algorithm whereby we trained 20 agents simultatiously. But instead of training those 20 agents, we only need to train 2 agents and those agents have to compete against each other.

Main Challenge?¶

The main challenge is not to implement the DDPG algorithm (that is something which we've proven earlier) But the main challenge now is, to compete 2 agents against each other!

DDPG with 2 collaborated agents - general overview¶

1) Environemnt: Reset the environment and provide 2 starting states back.
2) Agents: Ask each agent, given their state, which action would they perform
2.1) Provide the current state to the agents local actor network and receive an vector with length 2 back where values are between -1 and 1
3) Environemnt: perform the steps, suggested by the agents.
3.1) receive new states, reward and a done value
4) Each Agent: Perform a step
4.1) Add the state to the memory buffer
4.2) If there are enough data samples for creating a batch, we can train our networks
4.2.1) get a batch (states, actions, rewards, next_states, dones)
4.2.2) Predicted next-actions from the actor_target network (given next_state and agent) and Q-values from critic_target network (given next_states and next-actions)
4.2.3) Compute Q targets = rewards + (gamma Q_targets_next (1 - dones))
4.2.4) Compute the critic (MSE) loss between Q expected (result if critic_local(states, actions)) and Q_targets
4.2.5) Minimize the losses (make sure you use the correct actions) 4.2.6) slowly update target networks 5) Store the rewards and go to step 2 or quite when an agents is done :)

Actor¶

The Actor maps each state of the environment to an action, which is a vector of two numbers between -1 and +1. The network consists of three fully-connected linear layers with ReLU activation functions. The output layer uses the tanh activation function to produce values between -1 and +1 for each of the four outputs.

The task of the actor is to approximate the optimal policiy, deterministically. This means that we always want to output the best believed action for any given state.

Cretic¶

The Critic maps a state and action to a Q-value, which reflects the estimated quality of the given action in the given state. The network consists of three fully-connected linear layers with ReLU activation functions. The output layer produces the estimated value of the given action in the given state.

The task of the Cretic here is to approximate the maximizer over the Q values of the next state and not as a learned baseline.
The Cretic learns to evaluate the optimal value function by using the actors best believe action.

learn¶

4 networks

actor_local
actor_target
cretic_local
cretic_target

Gradient descent networks

actor_local
cretic_local

Slow update networks, via Tau

actor_target
cretic_target

Network which decides the next step

actor_local

Learn step

Grep a single Batch

GAMMA = 0.99
        experiences = memory.sample()

        # Unpack a single batch 
        states, actions, rewards, next_states, dones = experiences

Calculate the Q-target. This is done via the target-actor-network and target-cretic-network whereby we feed the outcome of the target-actor-network in the target-cretic-network, which results in a Q-target-next value.

Via the the Q-target-next value, we can calculate the Q-target.

# ---------------------------- update critic ---------------------------- #
        # Get predicted next-state actions and Q values from target models
        actions_next = self.actor_target(next_states)

        # create the correct actions next tuple based on the active agent !
        if agent_id == 0:
            actions_next_tuple = (actions_next, actions[:, self.ACTION_SIZE:])
        else:
            actions_next_tuple = (actions[:, :self.ACTION_SIZE], actions_next)

        actions_next = torch.cat(actions_next_tuple, dim=1)


        Q_targets_next = self.critic_target(next_states, actions_next)

        # Compute Q targets for current states (y_i)
        Q_targets = rewards + (gamma * Q_targets_next * (1 - dones))

Use the cretic_local-network with the state and action to calculate the MSE loss, with the Q-target (which were more based on the Next-state.

# Compute critic loss
        Q_expected = self.critic_local(states, actions)
        critic_loss = F.mse_loss(Q_expected, Q_targets)

Perform the gradient descent on the cretic-local-network

# Minimize the loss
        self.critic_optimizer.zero_grad()
        critic_loss.backward()
        self.critic_optimizer.step()

Calculate the actor-local loss

# ---------------------------- update actor ---------------------------- #
        # Compute actor loss
        actions_pred = self.actor_local(states)

        # create the correct actions_pred_tuple based on the active agent! 
        if agent_id == 0:
            actions_pred_tuple = (actions_pred, actions[:, self.ACTION_SIZE:])
        else:
            actions_pred_tuple = (actions[:, :self.ACTION_SIZE], actions_pred)


        actor_loss = -self.critic_local(states, actions_pred).mean()

Perform the gradient descent on the actor-local-network

# Minimize the loss
        self.actor_optimizer.zero_grad()
        actor_loss.backward()
        self.actor_optimizer.step()

Soft update the actor-target-network and cretig-target-network, via the hyper parameter TAU

# ----------------------- update target networks ----------------------- #
        self.soft_update(self.critic_local, self.critic_target, self.TAU)
        self.soft_update(self.actor_local, self.actor_target, self.TAU)

Agent¶

Load the agent class

from agent import Agent

Train¶

Create a train funcion

def run_ddpg(agents, n_episodes = 5000, max_t=10000, print_every=100, save_every=500, checkpoint_name=f'checkpoint_{get_time()}'):
    

    states_agent_size = num_agents * state_size  # size of the state space shared by the agents  
    
    scores_deque = deque(maxlen=print_every)
    scores = []
    
    # for each episode
    for i_episode in range(1, n_episodes+1):
        
        # start a new episode
        environment_info = env.reset(train_mode=True)[brain_name]
        states = environment_info.vector_observations.reshape(1, states_agent_size)
        
        for agent in agents:
            agent.reset()
            
        score = np.zeros(num_agents)
        
        # for max t steps in the episde
        for t in range(max_t):
            
            # get the actions from the agent, given a state
            actions = [agent.action(states, True) for agent in agents]
            actions =  np.hstack(tuple(actions))
    
            
            # perform the action & receive the next state, reward and done
            env_info = env.step(actions)[brain_name]           
            next_states = env_info.vector_observations.reshape((1, states_agent_size))
            
            rewards = env_info.rewards
            dones = env_info.local_done
 
            # learn if the critereia are met
            for i, agent in enumerate(agents):
                agent.step(t, states, actions, rewards[i], next_states, dones[i], i)
            
            # prepare next state + save results
            states = next_states      
            score += rewards
            
            # if any of the agents is done, we stop :) 
            if np.any(dones):
                break
                
        # store the scores | Log
        mean_score = np.mean(score)
        
        # add to the queue
        scores_deque.append(mean_score)
        scores.append(mean_score)
        
        # show logs
        avg_score = np.mean(scores_deque)
        wandb.log({"Average Score": avg_score, "Mean Score": mean_score})
        
        print(f'\rEpisode {i_episode}\tAverage Score: {np.around(avg_score,3)}, score: {np.around(mean_score,3)}', end="")
        

        if i_episode % save_every == 0:
            agent.save_agent(f'{checkpoint_name}_{i_episode}')
        

        if i_episode % print_every == 0 or avg_score >= 0.5:
            print(f'\rEpisode {i_episode}\tAverage Score: {np.around(avg_score,3)}, score: {np.around(mean_score,3)}')
        
        if avg_score >= 1.0:
            print(f'\r\nEnvironment is solved Episode {i_episode}\tAverage Score: {np.around(avg_score,3)}, score: {np.around(mean_score,3)}')
            agent.save_agent(f'{checkpoint_name}_{i_episode}')
            break
        
            
    return avg_score

hyper parameters¶

lr_actor : learning rate of the actor
lr_critic : learning rate of the critic
gamma : discount factor
weight_decay : L2 weight decay for the critic
tau : parameter for soft update of target parameters
learn_interval : learning interval
learn_num : Amount of batches which we would like to learn (one after the other)
batch_size : number of episodes which we would like to use to train
buffer_size : number of episodes from which we can can sample of

import json
with open('hyper_parameters.json') as json_file:
    hyper_parameters = json.load(json_file)

# show the parameters
hyper_parameters

{'random_seed': 0,
 'learn_interval': 2,
 'learn_num': 1,
 'lr_actor': 0.0007308,
 'lr_critic': 0.0009657,
 'gamma': 0.99,
 'weight_decay': 0,
 'tau': 0.0283,
 'batch_size': 256,
 'buffer_size': 100000}

import BayesianOptimization¶

In this project, i tried to automate the hyper parameters by using some bayesian optimization ...
Eventually, one of the initial random hyperparamters gave me one of the best results ...
(I think that I'm too impatient to discover the whole searchspace :-) ) but anyway a random search is often also a good solution :)

from bayes_opt import BayesianOptimization

The search space of the hyper parameters

bounds = {
    'learn_interval':(1,8),
    'lr_actor': (0.0005,0.003),
    'lr_critic': (0.0005,0.003),
    'tau':(0.001, 0.08)
}

An train model function, which is just a wrapper.
For each round, the hyperparameters come in and we collect them in a dictionary.
Then we load the initial hyperparameters and update this dictionary with the new hyperparameters
Last but not least, we create 2 agents, create a new wandb track and train the model.

def train_model(learn_interval, lr_actor, lr_critic, tau):
    
    params = {
        'learn_interval': int(learn_interval),
        'lr_actor': lr_actor,
        'lr_critic': lr_critic,
        'tau':tau
    }
    
    with open('hyper_parameters.json') as json_file:
        hyper_parameters = json.load(json_file)

    # show the parameters
    hyper_parameters.update(params)
    
    # create the agents
    agents = [Agent(state_size=state_size, action_size=action_size, **hyper_parameters) for _ in range(num_agents)]

    # Add the config to wandb    
    wandb.init(config=agents[0].__dict__,project="udacity_deep-reinforcement-learning_project-3")
    for agent in agents:
        wandb.watch(agent.actor_local)
        wandb.watch(agent.critic_local)
        
    # run reinforcement learning
    return run_ddpg(agents)

optimize¶

Run the Bayesian Optimization, we want to maximize the train model

optimizer = BayesianOptimization(
    f=train_model,
    pbounds=bounds,
    random_state=1,
)
optimizer.maximize(init_points=15, n_iter=50,)

|   iter    |  target   | learn_... | lr_actor  | lr_critic |    tau    |
-------------------------------------------------------------------------

INFO:wandb.run_manager:system metrics and metadata threads started
INFO:wandb.run_manager:checking resume status, waiting at most 10 seconds
INFO:wandb.run_manager:resuming run from id: UnVuOnYxOmVldW9semQyOnVkYWNpdHlfZGVlcC1yZWluZm9yY2VtZW50LWxlYXJuaW5nX3Byb2plY3QtMzp2ZXJiZWVtZW4=
INFO:wandb.run_manager:upserting run before process can begin, waiting at most 10 seconds
INFO:wandb.run_manager:saving patches
INFO:wandb.run_manager:saving pip packages
INFO:wandb.run_manager:initializing streaming files api
INFO:wandb.run_manager:unblocking file change observer, beginning sync with W&B servers

Episode 4	Average Score: -0.005, score: -0.005

INFO:wandb.run_manager:file/dir modified: /home/workspace/wandb/run-20191209_160812-eeuolzd2/config.yaml

Episode 7	Average Score: -0.005, score: -0.005

INFO:wandb.run_manager:file/dir created: /home/workspace/wandb/run-20191209_160812-eeuolzd2/wandb-history.jsonl
INFO:wandb.run_manager:file/dir created: /home/workspace/wandb/run-20191209_160812-eeuolzd2/requirements.txt
INFO:wandb.run_manager:file/dir created: /home/workspace/wandb/run-20191209_160812-eeuolzd2/wandb-events.jsonl
INFO:wandb.run_manager:file/dir created: /home/workspace/wandb/run-20191209_160812-eeuolzd2/wandb-metadata.json
INFO:wandb.run_manager:file/dir created: /home/workspace/wandb/run-20191209_160812-eeuolzd2/code/Tennis.ipynb
INFO:wandb.run_manager:file/dir created: /home/workspace/wandb/run-20191209_160812-eeuolzd2/wandb-summary.json
INFO:wandb.run_manager:file/dir created: /home/workspace/wandb/run-20191209_160812-eeuolzd2/code

Episode 15	Average Score: -0.005, score: -0.005

INFO:wandb.run_manager:file/dir modified: /home/workspace/wandb/run-20191209_160812-eeuolzd2/wandb-history.jsonl
INFO:wandb.run_manager:file/dir modified: /home/workspace/wandb/run-20191209_160812-eeuolzd2/wandb-summary.json

Episode 17	Average Score: -0.002, score: -0.005

INFO:wandb.run_manager:file/dir modified: /home/workspace/wandb/run-20191209_160812-eeuolzd2/wandb-history.jsonl
INFO:wandb.run_manager:file/dir modified: /home/workspace/wandb/run-20191209_160812-eeuolzd2/wandb-summary.json

Train for good !¶

After training differnt kind of models we can also train a single model

# create the agents
agents = [Agent(state_size=state_size, action_size=action_size, **hyper_parameters) for _ in range(num_agents)]

# Add the config to wandb    
wandb.init(config=agents[0].__dict__,project="udacity_deep-reinforcement-learning_project-3")
for agent in agents:
    wandb.watch(agent.actor_local)
    wandb.watch(agent.critic_local)


scores = run_ddpg(agents)

INFO:wandb.run_manager:system metrics and metadata threads started
INFO:wandb.run_manager:checking resume status, waiting at most 10 seconds
INFO:wandb.run_manager:file/dir created: /home/workspace/wandb/run-20191210_125114-bkisue6q/code/Tennis.ipynb
INFO:wandb.run_manager:resuming run from id: UnVuOnYxOmJraXN1ZTZxOnVkYWNpdHlfZGVlcC1yZWluZm9yY2VtZW50LWxlYXJuaW5nX3Byb2plY3QtMzp2ZXJiZWVtZW4=
INFO:wandb.run_manager:upserting run before process can begin, waiting at most 10 seconds
INFO:wandb.run_manager:saving patches
INFO:wandb.run_manager:saving pip packages
INFO:wandb.run_manager:initializing streaming files api
INFO:wandb.run_manager:unblocking file change observer, beginning sync with W&B servers
INFO:wandb.run_manager:file/dir created: /home/workspace/wandb/run-20191210_125114-bkisue6q/wandb-metadata.json
INFO:wandb.run_manager:file/dir created: /home/workspace/wandb/run-20191210_125114-bkisue6q/code
INFO:wandb.run_manager:file/dir modified: /home/workspace/wandb/run-20191210_125114-bkisue6q/config.yaml
INFO:wandb.run_manager:file/dir created: /home/workspace/wandb/run-20191210_125114-bkisue6q/wandb-events.jsonl
INFO:wandb.run_manager:file/dir created: /home/workspace/wandb/run-20191210_125114-bkisue6q/requirements.txt
INFO:wandb.run_manager:file/dir created: /home/workspace/wandb/run-20191210_125114-bkisue6q/wandb-history.jsonl
INFO:wandb.run_manager:file/dir created: /home/workspace/wandb/run-20191210_125114-bkisue6q/wandb-summary.json

Episode 9	Average Score: -0.005, score: -0.005

INFO:wandb.run_manager:file/dir modified: /home/workspace/wandb/run-20191210_125114-bkisue6q/wandb-history.jsonl
INFO:wandb.run_manager:file/dir modified: /home/workspace/wandb/run-20191210_125114-bkisue6q/wandb-summary.json

Episode 11	Average Score: -0.005, score: -0.005

INFO:wandb.run_manager:file/dir modified: /home/workspace/wandb/run-20191210_125114-bkisue6q/wandb-history.jsonl

Episode 12	Average Score: -0.005, score: -0.005

INFO:wandb.run_manager:file/dir modified: /home/workspace/wandb/run-20191210_125114-bkisue6q/wandb-summary.json

Episode 15	Average Score: -0.005, score: -0.005

INFO:wandb.run_manager:file/dir modified: /home/workspace/wandb/run-20191210_125114-bkisue6q/wandb-history.jsonl
INFO:wandb.run_manager:file/dir modified: /home/workspace/wandb/run-20191210_125114-bkisue6q/wandb-summary.json

Episode 18	Average Score: -0.005, score: -0.005

Results¶

Average score¶

Mean score¶

Evaluation¶

results It was written in the exercise that reaching the goal wouldn't be easy, and that statement is correct :).
Overall we managed to receive an average score of +0.5 twice, but we also tried +75 different hyperparameter combinations!
Also good to mention, we received multiple times a score of 2.5, but it seems to be that the current setup isn't that ... stable :)

hyperparameters The hyperparameters are very fragile (see the weights and biases report),
changing them a tiny bit can already immediately make a big difference.
Like said before, i've trained over +75 models and even with the bayesian optimization is getting a good result not that easy.

Future work¶

Optimize the network architectures: Both networks are quite shallow at this moment.
Try to implement other algorithms as wel, to see if they perform better or not
- D4PG (might before better then DDPG)
- A3C
- PPO